SOFTWARE RELIABILITY MODELING - 
WHERE ARE WE AND WHERE SHOULD WE BE GOING? 


THE NEED FOR SOFTWARE RELIABILITY MODELING 

It may be argued that software reliability metrics are needed, most importantly, because no field 
can really mature until it can be described in a quantitative fashion. However, there are also some 
very specific reasons for a quantitative approach to software reliability. One needs software 
liability figures in order to do a good job of system engineering: to examine the trade offs between 
reliability and cost and reliability and schedules, to determine what reliability figure optimizes 
overall life cycle costs, to plan allocation of resources, and to specify reliability to a contractor who 
is developing software for you. Another large area of application is project management, where 
software reliability measures are needed for progress monitoring, scheduling and investigation of 
managerial alternatives. The length of a test period and hence the overall length of a project is 
highly correlated with the reliability requirements for the project. Therefore, reliabilities are in- 
timately tied up with schedules. Oranges in resources available to the project affect both reliability 
and schedules and one can be exchanged for the other. Reliability metrics offer an excellent means 
of evaluating the performance of operational software and controlling changes to it. Since change 
usually involves a degradation of reliability, one may use reliability performance objectives as a 
means for determining when software changes can be allowed and perhaps even how large they 
can be. Finally, reliability is one of the important parameters that should be used in investigating 
iiie benefits (or lack of benefits) of proposed new software engineering technology. 


SOFTWARE RELIABILITY FUNCTIONS 

Hecht [ 1 ] has categorized software reliability functions into measurement, estimation and pre- 
diction. This classification is used in this paper with some modification and extension. Software 
reliability is defined as the probability that a program will execute without failure caused by 
software for a specified time in a specified environment. The term “failure” refers to an un- 
acceptable departure from proper operation. -The term “unacceptable” must be defined by the 
customer. The “measurement” of software reliability is based on failure interval data obtained by 
running the program in its actual operating environment. Software reliability “estimation” refers 
to the process of determining software reliability metrics based on operation in a test environment. 
It should be noted that estimation can be performed with respect to present or future reliabiUty 
quantities. The term software reliability “prediction” refers to the process of computing software 
reliability quantities from program data which does not include failure intervals. Typically, soft- 
ware reliability prediction takes into account factors such as size and complexity of the program, 
and is normally performed during a program phase prior to test. Note that future estimation 
might be thought of by some as prediction; we are deliberately making a careful distinction in 
terminology. , ’ 

The various applications of software reliability metrics are closely tied to the three functions that 
have just been defined. System engineering primarily relies upon prediction; project management, 
upon estimation; and operational software management and evaluation of software engineering 
technology, upon measurement. 
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SOFTWARE RELIABILITY MODELS 


Most of the work that has been done in the field of software reliability falls in one of six cate- 
gories: calendar time models, the execution time model, Bayesian models, semi-Markov models, 
deterministic models and input space approaches. The initial approach to software reliability was 
through calendar time models; that is, attempts were made to look of reliability phenomena such 
as failures, reliability, mean-time-to-failure (MTTF), etc. as functions of calendar time. These 
early models focused attention on the problem of software reliability and contributed many valu- 
able concepts toward the further development of the theory. [2-5] 

However, the failure-inducing stress placed oh software is related closely to execution time (CPU 
time) and not calendar time. The execution time model [6-11] recognizes this fact. It has been 
extensively tested on more than 20 software systems and the validity of the assumptions made in 
deriving the model has been carefully examined. [12] 

Littlewood and Verrall [13] have proposed a Bayesian model that is perhaps the most mathe- 
matically elegant of the software reliability models, but it is, unfortunately, difficult to understand, 
and computations based on it are lengthy and costly. A model that focuses specifically on the 
problem of imperfect fault correction has been developed by Goel and Okumoto [14] ; it is based 
on a view of fault correction as a semi-Markov process. The concept of imperfect fault correction 
is incorporated in the execution time model in a simpler fashion. Deterministic models have been 
proposed [15, 16] but they have not been validated. 

It would appear that deterministic models oversimplify the failure detection and correction process 
and are not efficient in using the information available to them. Bayesian models perhaps represent 
the other extreme, in that both failure intervals and failure process parameters are viewed as being 
random. The execution time model takes the intermediate approach of considering failure inter- 
vals random but failure process parameters as varying with execution time in a deterministic 
fashion. 

A final viewpoint, the input space approach, is based on enumerating all the possible sets of input 
or environmental conditions for a program and determining the proportion of these that result in 
successful operation. Although this approach is theoretically appealing, the large number of pos- 
sible input sets for any useful program makes it impractical. Tire counts would have to be weighed 
by run times and frequencies of operation for the various input sets, in order to provide results 
that would be compatible with haidware reliability theory. 


EXECUTION TIME MODEL 

The execution time model permits the development of relationships that indicate number of fail- 
ures experienced and present MTFF as functions of execution time (see Figures 1 and 2). It re- 
lates total failures and initial MTFF to the number of faults in the system. An initial estimate of 
the number of faults, prior to testing, can be determined from the size (and perhaps complexity) 
of the program. A debugging process model is provided which relates execution time and calendar 
time and thus allows execution time quantities to be converted into dates. The model can be 
used to make predictions of the remaining number of failures to be experienced, the execution 
time and the calendar time required to reach a MTTF objective. If this objective is set as the 
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criterion for terminating the project, completion dates can be predicted. As testing proceeds, two 
of the key parameters of the model can be statistically reestimated from failure intervals experienced. 
This permits the estimation of a number of derived quantities such as present MTTF and estimated 
completion date. The estimates made are maximum likelihood estimates; confidence intervals are 
also calculated. 

Most of the assumptions that were made in deriving the execution time model have been validated 
[12] and experience has been gained with the model on a wide variety of software systems (more 
than 20 as of this date). A program is available [17, 18] to handle the statistical calculations. 

Sample output from the program is shown in Figure 3. 

User comments indicate that the execution time model provides a good conceptual framework for 
viewing the software failure process. It is simple, its parameters are closely related to the physical 
world and it is compatible with hardware reliability theory. Most users feel that the benefits 
currently exceed the costs, which are basically data collection and computation. There have been 
two interesting side benefits. The process of defining just what constitutes a failure and the 
process of setting a MTTF objective have both been salutary in opening up communication be- 
tween customer and developer. 


STATE OF THE ART AND RESEARCH NEEDS 

Software measurement can presently be achieved with excellent accuracy. Figure 4 illustrates a 
software system in the operational state. The maximum likelihood estimate and 75% confidence 
bounds are indicated for present MTTF. Variations in MTTF and the size of the confidence inter- 
val are generally highly correlated with periods of fault correction or the addition of new 
capabilities. 

The quality of software reliability estimation is dependent upon the representativeness of testing; 
hence good test planning is esesntial. If one desires to know the absolute value of the MTTF, 
knowledge of the test compression factor is necessary. The test compression factor relates the 
amount of time spent in test with the equivalent amount'of operating time represented. It is 
known theoretically how to compute this number but the only practical approach at present is to 
estimate it from a similar ponect in a similar test environment. Research activity in this area 
would definitely be beneficial. One might characterize the present quality of software reliability 
estimation as good for present estimation and fair for future estimation. Future estimation also 
requires, in addition to the factors previously listed, a number of resource parameters. Data col- 
lection to determine the values of these parameters and the extent to which they vary between 
different projects or different classes of projects is urgently needed. Figure 5 illustrates the 
variation in present MTTF as the system test phase of a project proceeded (maximum likelihood 
estimate and 75% confidence bounds are indicated). Although the accuracy of the absolute es- 
timates is dependent on the test compression factor, the relative values (i.e.. denoting progress) 
are highly accurate. 

The function of software reliability prediction needs the most work. However, it also offers great 
promise in terms of ultimate potential benefits [9]. All of the input quantities required for soft- 
ware estimation are needed for this function as well. In addition, one requires the number of 
faults inherent in the software, the fault exposure ratio, the fault reduction factor and the linear 
execution frequency. Figure 6 indicates the quantities and relationships involved in software 
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reliability perdiction. The number of faults inherent in the software Nq must be determined from 
estimates of program size and data on fault densities. Data on fault densities is just beginning to 
accumulate but much more is needed, along with information on the variation of the fault density 
with program comple.xity and other factors. The fault reduction factor B indicates the ratio of net 
faults repaired to failures detected. It is a function of the test or operational environment and 
appears to be constant across similar environments. The initial MTTF, Tg, must be predicted from 
total failures Mg, from the linear execution frequency of the program f (throughput divided by 
object program size) and the fault exposure ratio K. The fault exposure ratio is expected to be 
dependent on the dynamic structure of the program and the degree to which faults are data de- 
pendent. Further investigation of the properties of this ratio and the factors upon which they de- 
pend is very important if we are to obtain good absolute software reliability predictions. Relative 
predictions can be made without this knowledge in many cases and they may be useful for many 
system engineering studies. 


CONCLUSIONS 

Software reliability has come a long way since its early beginnings in 1972. Many of the early 
problems have been solved and a reasonable amount of actual failure data has been collected. It 
may be seen from this paper that a number of problems remain to be solved and that new prob- 
lems will probably suggest themselves as the field progresses. It is important, however, that we 
build upon the results that have already been achieved so as to maximize the efficiency of our 
efforts. 
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FIGURE 2. 


EXECUTION TIME MODEL RELATIONSHIP. 
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Figure 3. Sample Output from Software Reliability Measurement/ 
Fstimation Program lor Hxeeution Time Model 


246 


J. Musa 
Bell Lab. 
8 of 11 





EST. PRESENT MTTF (HR) 


PROJECT 1 



0.01 LL- -J— 1 ' ' — -I 1 

7/73 8/73 9/73 10/73 11/73 

CURRENT DATE 


f'iuLirc 5. Software Reliability Estimation 


248 


J. Musa 
Bell Lab. 
10 of 11 



J. Musa 
Bell Lab 


SOFTWARE RELIABILITY PREDICTION 
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